If your prior is uniform, it applies the same multiplier to each possible value of \(\theta\) (your given parameter). So…
\[
\text{Posterior} \propto \text{Likelihood}
\]
Uniform Priors
Pros:
“Objective” 🙄
let’s the “data speak for themselves”
easy to explain
sometimes gives results roughly equivalent to Frequentist estimates
Cons:
you often know something (non-regularizing)
non-invariance under re-parameterization
improper over \(\mathbb{R_n}\)
Uniform Priors
library(brms)# freqlm(y ~ x, data = df) |>summary()
Call:
lm(formula = y ~ x, data = df)
Residuals:
Min 1Q Median 3Q Max
-1.82484 -0.59236 -0.03289 0.64124 2.88284
Coefficients:
Estimate Std. Error t value Pr(>|t|)
(Intercept) 0.08791 0.09502 0.925 0.357
x -0.52576 0.09892 -5.315 6.7e-07 ***
---
Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
Residual standard error: 0.9501 on 98 degrees of freedom
Multiple R-squared: 0.2238, Adjusted R-squared: 0.2158
F-statistic: 28.25 on 1 and 98 DF, p-value: 6.703e-07
Uniform Priors (Freq)
# bayespriors <-c(prior("", class ="b"),prior("", class ="Intercept"),prior("uniform(0, 1e6)", class ="sigma"))brm(y ~ x, data = df, prior = priors, verbose =2) |>summary()
Family: gaussian
Links: mu = identity; sigma = identity
Formula: y ~ x
Data: df (Number of observations: 100)
Draws: 4 chains, each with iter = 2000; warmup = 1000; thin = 1;
total post-warmup draws = 4000
Regression Coefficients:
Estimate Est.Error l-95% CI u-95% CI Rhat Bulk_ESS Tail_ESS
Intercept 0.09 0.10 -0.10 0.28 1.00 4245 2895
x -0.52 0.10 -0.72 -0.33 1.00 4159 2776
Further Distributional Parameters:
Estimate Est.Error l-95% CI u-95% CI Rhat Bulk_ESS Tail_ESS
sigma 0.96 0.07 0.84 1.10 1.00 4471 3159
Draws were sampled using sampling(NUTS). For each parameter, Bulk_ESS
and Tail_ESS are effective sample size measures, and Rhat is the potential
scale reduction factor on split chains (at convergence, Rhat = 1).
Now, let’s look at what Parameterization 2 (uniform prior on \(e^\beta\)) implies about \(\beta\):
Jeffrey’s Prior
\[
p(\theta) \propto \sqrt{I(\theta)}
\]
Jeffrey’s Prior: method of constructing non-informative priors that are invariant under re-parameterization
(see Bayesian Data Analysis 2.8 for more math)
Fisher Information
Slightly Mathy Idea: Fisher Information measures how sensitive the log-likelihoodfunction \(\ell(\theta | X)\) is to changes in \(\theta\) (more sensitive \(\to\) more information)
where \(\ell(\theta | X)\) is the log-likelihood of \(\theta\) given \(X\). If \(\ell\) is sensitive to changes in \(\theta\), the second derivative should be large and we expect to see high information
Note: usually \(\ell\) is concave down around maximum likelihood estimate, meaning that the second-derivative will be negative, hence the negative sign in front of the expectation
Uniform Priors (Regularization)
Regularization (in general): discourages overfitting, makes models simpler
Regularizing Priors: keeps parameter values in a “reasonable range”
Uniform Priors (Regularization)
Ridge = Normal Prior, Lasso = Laplacian Prior.
Other Types of Priors
Uniformative: all possible values for \(\theta\) have the same relative prior likelihood
Weakly Informative: regularize our estiamtes, extreme values are less likely, but not much other info
Strongly Informative: prior is narrow around known likely values
Choosing Prior Distributions
think about the range of possible values of \(\theta\)
think about amount of prior uncertainty
think about regularization
In reality, we often report 2-3 different versions of the models with various priors to see how these priors change the inferences we make.
Setting Priors in brms
set.seed(540)n <-100x <-rnorm(n)y <-0.25- x*0.5+rnorm(n)df <-data.frame(x,y)# bayespriors <-c(prior("normal(0,5)", class ="b"),prior("normal(0,4)", class ="Intercept"),prior("gamma(0.1, 10)", class ="sigma"))brm(y ~ x, data = df, prior = priors, verbose =2) |>summary()
Family: gaussian
Links: mu = identity; sigma = identity
Formula: y ~ x
Data: df (Number of observations: 100)
Draws: 4 chains, each with iter = 2000; warmup = 1000; thin = 1;
total post-warmup draws = 4000
Regression Coefficients:
Estimate Est.Error l-95% CI u-95% CI Rhat Bulk_ESS Tail_ESS
Intercept 0.25 0.10 0.06 0.44 1.00 4512 2874
x -0.49 0.11 -0.70 -0.28 1.00 4252 2718
Further Distributional Parameters:
Estimate Est.Error l-95% CI u-95% CI Rhat Bulk_ESS Tail_ESS
sigma 0.98 0.07 0.87 1.13 1.00 3547 2877
Draws were sampled using sampling(NUTS). For each parameter, Bulk_ESS
and Tail_ESS are effective sample size measures, and Rhat is the potential
scale reduction factor on split chains (at convergence, Rhat = 1).
